Skip to content

Review PRs larger than the size gate: diff-only and partial review, raise command, masked lockfiles - #34

Merged
irvingpop merged 7 commits into
mainfrom
irvingpop/review-prs-larger-than-the-size-gate
Nov 14, 2026
Merged

irvingpop merged 7 commits into
mainfrom
irvingpop/review-prs-larger-than-the-size-gate

Conversation

@irvingpop

Copy link
Copy Markdown
Member

Closes #29.

A PR over the size limit now gets a model review instead of a skip.

bundle -> over 16 MiB? --yes--> skipped (existing reason)
  | no
  v
fits limit? --yes--> full review
  | no
  v
drop full-file content: added -> docs -> tests -> largest   (every patch stays)
  | fits? --yes--> diff-only review (summary lists the files)
  | no
  v
patches by priority up to the limit, cut only between files
  -> partial review, gate fails with waivable "partial model review", names unseen files

What changed

  • Size limit in real tokens. Estimate is chars/1.7. Base limit 200,000. 4x = 800,000 stays under the 1M window with prompts and max_tokens.
  • Raise command. /oc-review size-limit 2x|3x|4x, write or admin only (PR author too, if they have write). Bound to the PR, survives force pushes. Newest valid raise wins. Rejected raises show in the summary comment with the reason. Override parser now skips size-limit comments.
  • Lockfiles (owner comment on Review PRs larger than the size gate #29, redesigned). Lockfiles never send full content. The patch always stays, with hash values cut to 8 chars + ~. Nothing inside a URL is masked. If the mask makes a removed and an added line equal, both show unmasked. dependency_delta gets removed rows. The first idea (replace the patch with a pointer to the delta) was dropped: the delta misses -r <url>, --trusted-host, integrity-only and install-script-only changes.
  • Citation check accepts quotes from the masked lockfile text.
  • Ratio log. Publisher JSON line has the estimate next to the real input usage per pass.
  • Spec: Bundle, Reasons, Override, Auth and cost, Publishing, Testing, decision log P6-6.

Testing

  • cd review && uv run pytest tests/unit: 3324 passed, 2 deselected.
  • Collect-only run on Bump opentelemetry-api from 1.45.1 to 1.45.2 in /review #33 (GitHub REST reads, no Messages API call): full mode, about 5.5k estimated tokens. uv.lock sent no content, all 6 changed lines kept, hashes masked.
  • Reviews: spec and each code task by a fresh reviewer, final adversarial review of the whole change. Found and fixed: unsafe lockfile patch removal, mask hiding URL data, three quadratic scans in the mask that an outside contributor could use to time out collect.

Not done

  • The ~800k-token long-context eval. It costs money and needs owner approval.
  • Live check of the comment and collaborator-permission reads from the collect job token (a failed read falls back to the base limit, it does not fail the run).

Known residuals

  • The mask still cuts one hash-shaped token in rare URL forms: a tab or space inside a URL, https:/host or https:\host, schemes with no // (github:, git@host:). The host stays visible.
  • An author can steer which files lose full content by path and size (accepted in Review PRs larger than the size gate #29). Every changed line stays in view.
  • New timing regressions use a 5 s bound. They could flake on a very slow runner.

🤖 Generated with Claude Code

irvingpop and others added 5 commits November 14, 2026 09:05
…ssue 29)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ckfile patches (issue 29)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tations, size log keys (issue 29)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@irvingpop
irvingpop requested a review from a team as a code owner November 14, 2026 18:11
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

🔴 Hold deployment: Claude of Duty identified 3 items requiring attention

Mission: Review the code. Protect the branch. Deploy the future.

Caution

Incomplete file coverage and 1 other hard block(s) block this PR. No waiver can clear them.
Next step for the author: fix the cause named in the row, then push. The review runs again on the new commit.

Severity Effect Where Finding Found by
🚫 Hard block Blocks, no waiver review/tests/unit/test_collect.py Incomplete file coverage: file content not scanned: unterminated-private-key, in review/tests/unit/test_collect.py File checks (code, no model)
🚫 Hard block Blocks, no waiver review/tests/unit/test_render.py Incomplete file coverage: file content not scanned: unterminated-private-key, in review/tests/unit/test_render.py File checks (code, no model)
⚠️ Not run Blocks, waivable Model passes Too large for automatic review. File checks ran, model checks did not. Workflow (code, no model)
File checks Review Security Supply chain Dependency review Coverage
🚫 2 hard blocks, 9 signals ⚠️ skipped, too large ⚠️ skipped, too large ⚠️ skipped, too large ✅ ok 🚫 incomplete

Run 43630465795, attempt 1 for head 5c34ab4 into main. The required workflow in the merge box is the result that counts.

Claude of Duty never approves a PR. A human reviewer gives the final approval.

📋 Full review record: reasons, checks, signals, finding details, model summaries, token usage

Reviewed snapshot

Head SHA Merge base SHA Base branch Run
5c34ab44596f0e609c3f712dcf2ab28598888d27 6b13e730d5b29927b258faef07478a447ccb3768 main 43630465795, attempt 1

Reasons the gate failed

Reason Waivable? Detail
Change too large for the model review ✅ yes too large for automatic review
Hard block from the file checks 🚫 no incomplete-coverage in review/tests/unit/test_collect.py
Hard block from the file checks 🚫 no incomplete-coverage in review/tests/unit/test_render.py
Hard block from the file checks 🚫 no incomplete-coverage (coverage record is not complete)

No waiver line is shown. At least one reason cannot be waived, so a waiver would not change the result.

Checks (code, no model)

Check Result Detail
Hard block 🚫 blocks Incomplete file coverage: file content not scanned: unterminated-private-key, in review/tests/unit/test_collect.py
Hard block 🚫 blocks Incomplete file coverage: file content not scanned: unterminated-private-key, in review/tests/unit/test_render.py
Coverage 🚫 NOT complete files in comparison 100, files collected 100, trees complete
Unavailable file ⚠️ unavailable review/tests/unit/test_collect.py: unterminated-private-key
Unavailable file ⚠️ unavailable review/tests/unit/test_render.py: unterminated-private-key
Dependency changes ✅ none
Dependency review job ✅ success

Signals (context only, never block)

  • ⚠️ Sensitive path changed: change to workflow, in .github/workflows/eval.yml
  • ℹ️ Text aimed at an AI reviewer: pattern system-prompt, in docs/claude-review-design.md, lines 263
  • ℹ️ Very long line: lines longer than 500 characters, in docs/claude-review-design.md, lines 212 249 261 263 305 306 313 352 359 367 and more
  • ℹ️ Very long line: lines longer than 500 characters, in review/prompts/system.md, lines 18
  • ℹ️ Hidden invisible character: invisible characters U+034F, U+115F, U+1160, U+3164, U+FFA0, in review/src/oc_review/prefilter.py, lines 122
  • ℹ️ Very long line: lines longer than 500 characters, in review/tests/fixtures-long-context/long-context-4x/bundle.json, lines 300 302 303 304 320 322 323 324 340 342 and more
  • ℹ️ Very long line: lines longer than 500 characters, in review/tests/fixtures/requirements-include-url/bundle.json, lines 44 45
  • ℹ️ Encoded blob: base64 run longer than 200 characters, in review/tests/unit/fixtures/schema/envelope/bad-bundle-size-long-reason.json, lines 48
  • ℹ️ Hidden zero-width character: zero-width characters U+200B, in review/tests/unit/test_verdict.py, lines 704 707

Dependency changes

None.

Finding details

Model output from untrusted input. Check it before you act on it. Verified means only that the quoted text is at that place. It does not mean the claim is true.

Blocking is decided by code from category and severity, never by the model. A finding in a blocking category blocks only at high or critical severity.

No findings from the passes that ran. The Review, Security, Supply chain pass did not run, so this is not a clean result.

Model summaries

Model output from untrusted input. Check it before you act on it. Verified means only that the quoted text is at that place. It does not mean the claim is true.

Pass Status Model Summary
Review ❌ skipped-size No summary. Reason: bundle status skipped-size
Security ❌ skipped-size No summary. Reason: bundle status skipped-size
Supply chain ❌ skipped-size No summary. Reason: bundle status skipped-size

Token usage

No token usage was recorded for any pass.

The reviewer's own bidi-control hard block fired on its detector regex,
so every PR that touched prefilter.py was blocked. Values are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@irvingpop
irvingpop merged commit c129aa9 into main Nov 14, 2026
6 of 7 checks passed
@irvingpop
irvingpop deleted the irvingpop/review-prs-larger-than-the-size-gate branch November 14, 2026 19:14

This branch was successfully deployed

1 active deployment
eval-test — 5c34ab44 Deployed Nov 14, 2026 by irvingpop via fixture-evals #20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Review PRs larger than the size gate

1 participant